in_tail: Implement long line truncation #11059
Merged
+375
−9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughAdds a configurable Changes
Sequence Diagram(s)sequenceDiagram
participant Reader as File Reader
participant Processor as in_tail Processor
participant Truncator as UTF-8 Truncator
participant Metrics as Metrics Registry
participant Output as Output Queue
Note right of Reader: read chunk bytes
Reader->>Processor: provide chunk
Processor->>Processor: decode/convert (ret)
Processor->>Processor: compute eff_max / search newline
alt newline found within window
Processor->>Output: emit complete line
else truncate_long_lines enabled and dec_len >= eff_max
Processor->>Truncator: find safe cut position
Truncator->>Processor: cut index
Processor->>Output: emit truncated segment
Processor->>Metrics: increment long_line_truncated
Processor->>Processor: set skip_next, adjust bytes/offsets
else truncate_long_lines disabled
Processor->>Processor: skip/drop until newline (existing behavior)
end
Processor->>Reader: return processed byte count / update DB offsets
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes
Possibly related PRs
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (31)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
cosmo0920
force-pushed
the
cosmo0920-implement-long-line-truncation
branch
from
dbb5d14 to
3119662
Compare
Member
|
@cosmo0920 we need commit history cleanup to get this merged |
Signed-off-by: Hiroshi Hatake <[email protected]>
cosmo0920
force-pushed
the
cosmo0920-implement-long-line-truncation
branch
from
903bcbf to
8d2cf71
Compare
Contributor
Author
|
Yup. I cleaned up my commit history into 2 commits which are an implementation and its unit test. |
Signed-off-by: Hiroshi Hatake <[email protected]>
cosmo0920
force-pushed
the
cosmo0920-implement-long-line-truncation
branch
from
8d2cf71 to
80ac24c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For avoiding to skip long line consumption,
it sometimes needs to consume until the limit of buffers. This could provide different approach of mitigation for consuming long lines.
Fixes #10435.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
New Features
Bug Fixes
Tests